Proximity detector in handheld device

ABSTRACT

Proximity based systems and methods that are implemented on an electronic device are disclosed. The method includes sensing an object spaced away and in close proximity to the electronic device. The method also includes performing an action in the electronic device when an object is sensed.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.14/080,116 (U.S. Patent Publication No. 2014-0074426), filed Nov. 14,2013, which is a continuation of U.S. application Ser. No. 13/766,570(now U.S. Pat. No. 8,612,856), filed Feb. 13, 2013, which is acontinuation of U.S. application Ser. No. 11/240,788 (now U.S. Pat. No.8,381,135), filed Sep. 30, 2005, which is a Continuation-in-Part of U.S.patent application Ser. No. 10/903,964 (now U.S. Pat. No. 8,479,122),filed Jul. 30, 2004, and is a Continuation-in-Part of U.S. patentapplication Ser. No. 11/038,590 (now U.S. Pat. No. 8,239,784), filedJan. 18, 2005, and is a Continuation-in-Part of U.S. patent applicationSer. No. 10/927,925 (now U.S. Pat. No. 7,760,187), filed Aug. 26, 2004,and is a continuation of U.S. patent application Ser. No. 11/165,958(now U.S. Pat. No. 7,599,044), filed Jun. 23, 2005, all of which areherein incorporated by reference.

This application is also related to U.S. Pat. No. 6,583,676, titled“PROXIMITY/TOUCH DETECTOR AND CALIBRATION CIRCUIT,” issued Jun. 24,2003; U.S. Pat. No. 7,663,607, titled “MULTIPOINT TOUCHSCREEN,” issuedFeb. 16, 2010; and U.S. Pat. No. 7,653,883, titled “PROXIMITY DETECTORIN HANDHELD DEVICE,” issued Jan. 26, 2010, all of which are hereinincorporated by reference.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates generally to I/O platforms includingproximity detection as a means for providing inputs. More particularly,the invention relates to improved methods and devices for operating aportable electronic device that utilizes proximity detection.

2. Description of the Related Art

There exist today many styles of input devices for performing operationsin an electronic system. The operations generally correspond to moving acursor and making selections on a display screen. By way of example, theinput devices may include buttons, switches, keyboards, mice,trackballs, touch pads, joy sticks, touch screens and the like. Each ofthese devices has advantages and disadvantages that are taken intoaccount when designing the electronic system.

Although devices such as these work well, there are continuing effortsto improve their form, feel and functionality. An improved input meansand operational methods associated therewith is therefore desired.

SUMMARY OF THE INVENTION

The invention relates, in another embodiment, to a method for initiatingfloating controls on an electronic device. The method includes detectingthe presence of an object above and spaced away from a surface of theelectronic device. The method also includes displaying a particulargraphical user interface element on a display of the electronic devicewhen the object is detected above the surface of the electronic device.

The invention relates, in another embodiment, to a user interfacemethod. The method includes detecting an object in space. The methodalso includes determining a user interface mode when an object isdetected in space. The method further includes displaying and enablingone or more GUI elements based on the user interface mode.

The invention relates, in another embodiment, to a method for scrollingthrough media items. The method includes displaying a plurality of mediaitems at a surface. The method also includes detecting an object spacedaway from the surface and over at least one of the media items that aredisplayed. The method further includes activating a virtual scroll wheelwhen the object is detected over the displayed media items. The methodadditionally includes detecting a touch on the surface and over thevirtual scroll wheel. The method also includes determining if a touchevent is performed on the surface and over the virtual scroll wheel.Moreover, the method includes scrolling through the group of media itemswhen a scrolling touch event is performed.

The invention relates, in another embodiment, to a computer implementedmethod. The method includes presenting graphical information on adisplay having a display surface. The method also includes detecting anobject in space above display surface and over the display. The methodfurther includes expanding an area of the graphical informationproximate the location of the object when the object is detected inspace above display surface and over the display.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention may best be understood by reference to the followingdescription taken in conjunction with the accompanying drawings inwhich:

FIG. 1 is a diagram of an I/O platform, in accordance with oneembodiment of the present invention.

FIG. 2 is a diagram of an I/O platform, in accordance with oneembodiment of the present invention.

FIG. 3 is a diagram of an I/O platform, in accordance with oneembodiment of the present invention.

FIG. 4 is a diagram of a method of operating an I/O platform, inaccordance with one embodiment of the present invention.

FIG. 5 is a simplified side elevation view of a portable computingdevice, in accordance with one embodiment of the present invention.

FIG. 6 is a diagram of a proximity based method, in accordance with oneembodiment of the present invention.

FIG. 7 is a diagram of a method of operating an electronic device, inaccordance with one embodiment of the present invention.

FIG. 8 is a diagram of a proximity method of operating an electronicdevice, in accordance with one embodiment of the present invention.

FIG. 9 is a diagram of a touch method, in accordance with one embodimentof the present invention.

FIG. 10 is user interface method, in accordance with one embodiment ofthe present invention.

FIG. 11 is a determination method, in accordance with one embodiment ofthe present invention.

FIG. 12 is a user interface method, in accordance with one embodiment ofthe present invention.

FIG. 13A-13F illustrates a sequence associated with the method shown inFIG. 12, in accordance with one embodiment of the present invention.

FIG. 14A-14F illustrates a sequence associated with the method shown inFIG. 12, in accordance with one embodiment of the present invention.

FIGS. 15A-15C illustrate a user interface sequence, in accordance withone embodiment of the present invention.

FIG. 16 is a user interface method, in accordance with one embodiment ofthe present invention.

FIGS. 17A-17J illustrate a user interface sequence, in accordance withone embodiment of the present invention.

FIG. 18 is a diagram of a GUI operational method, in accordance with oneembodiment of the present invention.

FIGS. 19A-19D illustrate a zooming target sequence using the GUIoperational method, in accordance with one embodiment of the presentinvention.

FIG. 20 is an expansion method, in accordance with one embodiment of thepresent invention.

FIG. 21 is a proximity method, in accordance with one embodiment of thepresent invention.

FIG. 22 is a proximity gesture method, in accordance with one embodimentof the present invention.

FIG. 23 is a block diagram of an exemplary computer system, inaccordance with one embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

Embodiments of this aspect of the invention are discussed below withreference to FIGS. 1-23. However, those skilled in the art will readilyappreciate that the detailed description given herein with respect tothese figures is for explanatory purposes as the invention extendsbeyond these limited embodiments.

FIG. 1 is a simplified block diagram of an I/O platform 10, inaccordance with one embodiment of the present invention. The I/Oplatform 10 may be embodied as a peripheral stand alone device or it maybe integrated into an electronic system or device. In either case, theI/O platform 10 is configured to control inputs and/or outputsassociated with the host system or device. Examples of peripheral standalone devices include mice, keyboards, joysticks, remote controls, etc.Examples of electronic devices include any consumer related electronicdevice such as computers, media players, telephones, etc.

The I/O platform 10 includes an I/O surface 12 having one or more inputand/or output devices 14. The input devices 14A may for example beselected from buttons, switches, dials, sliders, keys or keypads,navigation pads, touch pads, touch screens, and the like. The outputdevices 14B may for example be selected from displays (e.g., LCD, CRT),speakers (or speaker jacks), indicators, and the like. The I/O devices14 can be used solely or in any combination with other I/O devices 14 toform the desired I/O platform 10.

The I/O platform 10 also includes a proximity detection system 16configured to detect when a finger (or stylus) is in close proximity to(but not in contact with) the I/O surface 12. The proximity detectionsystem 16 may also detect location (e.g., x, y, z), direction, speed,orientation (e.g., roll, pitch, yaw), etc. of the finger relative to theI/O surface 12. Generally speaking, the proximity detection system 16provides additional inputs to the I/O platform 10 when the a finger isplaced above the I/O surface 12 as for example when a finger is movingdirectly towards the I/O surface 12, hovering just over the I/O surface12 or possibly moving about but above the I/O surface 12. The inputsignals can be used to initiate commands, make selections, and evencontrol motion in a display (e.g., tracking). In some cases, the inputsprovided by the proximity detection system 16 may even be used inconjunction with other inputs (provided by other input devices) tocreate new inputs, which have different meaning than when the two inputsare used by themselves. For example, when using a touch sensing devicealong with a proximity detection system, advanced gestures may beperformed that combine proximity gestures along with touch gestures.

The proximity detection system 16 may be widely varied. For example, itmay be based on sensing technologies including capacitive, electricfield, inductive, hall effect, reed, eddy current, magneto resistive,optical shadow, optical visual light, optical IR, optical colorrecognition, ultrasonic, acoustic emission, radar, heat, sonar,conductive or resistive and the like. A few of these technologies willnow be briefly described.

Capacitive proximity systems use capacitive sensing nodes to detectchanges in capacitance above the I/O surface. Objects such as a fingeraffect the capacitance in a non trivial manner and therefore the changescan be used to determine whether the finger is present above the I/Osurface.

Infrared proximity systems send out pulses of infrared light, and detectreflections of that light from nearby objects. The light may for examplebe reflected off of a finger located in front of the pulsed light. Ifthe system detects reflected light, then an object is assumed to bepresent. If the system does not detect reflected light, then it isassumed that there is no object present. The light may be focused to aparticular distance above the I/O surface.

Optical Shadow proximity systems use photodetectors or similar device todetect changes in light intensity. If the intensity is reduced, it canbe assumed that this was caused by the presence of an object forming ashadow over the photodetector.

Ultrasonic proximity systems provide ultrasonic pulses of sound, andmeasures the length of time it takes for those pulses to hit nearbyobjects and return as echos. The greater the time, the further theobject.

In most cases, the proximity detection system 16 includes one or moreproximity sensors 18 that generate one or more sensing fields 19 abovethe I/O surface 12 and that in conjunction with a controller 20 producesignals when an object disturbs or intercepts the sensing field(s). Eachsensing field typically generates its own signals when disturbed. In oneembodiment, a single sensing field is used to cover the entire I/Osurface. In another embodiment, a single sensing field only covers aportion of the I/O surface. For example, it can be configured to coveror surround a particular I/O device or a group of I/O devices. Inanother embodiment, multiple sensing fields are used to cover the entireI/O surface. For example a first sensing field may be configured tocover a first I/O device or a first group of I/O devices and a secondsensing field may be configured to cover a second I/O device or a secondgroup of I/O devices. Any number of sensing fields may be used. In somecases, in order to perform tracking, the sensing fields may even bedistributed as a pixilated array of nodes. In yet another embodiment,multiple sensing fields are used to only cover a portion of the I/Osurface. Alternatively or additionally, the sensing fields may beconfigured to cover areas that do not include I/O devices as for examplehousing components that support the I/O platform 10.

Although the I/O surface 12 can be widely varied, several embodimentswill now be discussed. It should be appreciated that these embodimentsare not limitations and that they serve as examples to the infinitenumber of combinations that can be created when designing an I/O surface12.

In one embodiment, the I/O surface 12 includes an output device in theform of a display and input devices in the form of a touchpad and one ormore buttons. By way of example, the display may be situated in an upperregion of the I/O surface, and the touch pad and buttons may be situatedin a lower region of the I/O surface just below the display. Any or allof these devices may include an all encompassing or regional sensingfield 19 disposed around and/or above the devices. This type of I/Osurface can typically be found on media players such as the iPod musicplayers manufactured by Apple Computer of Cupertino, Calif.

In another embodiment, the I/O surface 12 includes an enlarged touchscreen display, a navigation pad, and one or more buttons. By way ofexample, the enlarged display may be situated in an upper/middle regionof the I/O surface 12, and the navigation pad and buttons may besituated in a lower region of the I/O surface 12 just below the display.Any or all of these devices may include an all encompassing or regionalsensing field 19 disposed around and/or above the devices. This type ofI/O surface can typically be found on personal digital assistants(PDAs).

In another embodiment, the I/O surface 12 includes a small display,navigation pad and a keypad. By way of example, the small display may besituated in an upper region of the I/O surface 12, and the navigationpad and keypad may be situated in a lower/middle region of the I/Osurface 12 just below the display. Any or all of these devices mayinclude an all encompassing or regional sensing field 19 disposed aroundand/or above the devices. This type of I/O surface can typically befound on cellular phones.

In another embodiment, the I/O surface 12 only includes a touch screendisplay. That is, the entire I/O surface 12 is covered with a displaywith a touch screen disposed over the display. This type of I/O surface12 can be found in multifunctional handheld devices where the displaycovers an entire side of the device. Any or all of these devices mayinclude an all encompassing or regional sensing field 19 disposed aroundand/or above the devices. An example of a multifunctional device can befound in U.S. Provisional Patent Application No. 60/658,777, which isherein incorporated by reference.

In another embodiment, the I/O platform 12 is a peripheral stand aloneinput device such as a keyboard or mouse. The I/O surface 12 of thekeyboard includes a plurality of alphanumeric keys. The I/O surface 12of the mouse includes one or more buttons and possibly a scroll wheel.Any or all of these devices may include an all encompassing or regionalsensing field 19 disposed around and/or above the devices.

In another embodiment, the I/O platform 12 is a peripheral stand aloneinput output device such as a monitor that includes a touch screendisplay. Any or all of these devices may include an all encompassing orregional sensing field 19 disposed around and/or above the devices.

In another embodiment, the I/O platform 12 is a wireless peripheralstand alone input device such as a remote control. Remote controlstypically include a plurality of buttons and keys and may include adisplay. Any or all of these devices may include an all encompassing orregional sensing field disposed around and/or above the devices.

FIG. 2 is a diagram of an I/O platform 30, in accordance with oneembodiment of the present invention. The I/O platform 30 includes an I/Osurface 32 including any number of I/O devices 34 in any number ofcombinations. The I/O platform 30 also includes an IR sensing circuit 36for detecting the presence of an object 38 in close proximity but spacedapart from the I/O surface 32 (non contact). The IR sensing circuit 36works by sending pulses of light, and detecting the light when itreflects off of the object 38 above the I/O surface 32. The IR sensingcircuit 36 includes an emitter 40 (light source) and a detector 42. Theemitter 40 generates light in the infrared bands. The emitter 40 may forexample be an LED or laser diode. The detector 42, on the other hand, isconfigured to detect changes in light intensity. This is generallyaccomplished by mechanisms capable of varying electrically as a functionof light intensity. The detector 42 may for example be aphototransistor.

During operation, the light from the emitter 40 hits the object 38 andscatters when a object is present above the I/O surface 32. At least aportion of the scattered light is reflected towards the detector 42. Theincrease in light intensity is read by the detector 42, and this isinterpreted by a controller 44 to mean an object is present. If noobject is present, the light is not reflected back towards the detector42, and this is interpreted by the controller 44 to mean an object isnot present.

In one embodiment, the emitter 40 and detector 42 are disposed within ahousing 46 and underneath a window 48 of the housing 46. The window maybe formed from a translucent or semi translucent material. The windowmay be the entire surface or it may only be a portion of the housing 46.The emitter 40 and detector 42 may be located at any position within theI/O surface 32. In some cases, the emitter 40 and detector 42 arelocated at the edge (e.g., bezel), and angled inward towards one anotherto improve the detection. It should be appreciated, however, that thisis not a requirement. For example, they may also be positioned at morecentral locations and may even be positioned around a particular I/Odevice. Furthermore, although only one emitter 40 and detector 42 areshown, it should also be appreciated that this is not a limitation andthat multiple sets may be used. This may be done to improve detection(e.g., averaging) or to create a multiple sensing fields or nodes. Insome cases, an array of emitters and corresponding detector array may beused.

FIG. 3 is a diagram of an I/O platform 50, in accordance with oneembodiment of the present invention. The I/O platform 50 includes an I/Osurface 52 including any number of I/O devices 54 in any number ofcombinations. The I/O platform 50 also includes a capacitive sensingcircuit 56 for detecting the presence of an object 58 in close proximitybut spaced apart from the I/O surface 52 (non contact). The capacitivesensing circuit 56 works by detecting changes in capacitance in an area60 above the I/O surface 52. As should be appreciated, when an object(e.g., finger) is placed in the sensing area 60, the value of thecapacitance changes due to the fact that the finger has differentdielectric properties than air. Any changes can therefore be used todetermine the presence of the object 58 above the I/O surface 52.

The capacitive sensing circuit 56 may include one or more capacitivesensing nodes 62. The nodes 62 may be positioned within, around and/orunderneath the I/O devices 54 supported by the I/O surface 52.

The capacitive sensing nodes may be widely varied. For example, they maybe based on self capacitance or mutual capacitance. In self capacitance,each of the nodes 62 typically consist of a conductive electrode. Inmutual capacitance, each of the nodes 62 typically consists ofconductive wires lying crosswise relative to each other (the point wherethe wires cross is considered the node). In either case, the conductiveelement may be integrally formed into the housing walls 64 of the I/Oplatform 50 or they may be located behind the housing walls 64 of theI/O platform 50. The housing wall 64 may for example be the housing wallthat supports the I/O surface 52. Alternatively or additionally, theconductive elements may be integrated within the I/O devices themselves.Furthermore, they are typically coupled to a controller 66 that monitorsthe capacitance changes at each conductive element. By detecting changesin capacitance at each of the conductive elements, the microcontroller66 can determine the presence of a object 58 or lack thereof above theI/O surface 52.

In one embodiment, a single node may be configured to cover the entireI/O surface. For example, in the case of self capacitance, a singleelectrode may be spread across the entire I/O surface.

In another embodiment, a single node is configured to cover some subregion of the I/O surface. In one implementation of this embodiment, forexample, a single node may be configured to cover all the areas of thehousing wall 64 that surround the various I/O devices 54. In anotherimplementation, a single node is configured to cover all the areasunderneath the various I/O devices 54. In another implementation, asingle node may be positioned at one or more of the I/O devices 54rather than the entire set of I/O devices. These nodes may be positionedaround and/or underneath the I/O devices 54. In another implementation,a single node may be positioned around the edge of the I/O surface 52(e.g., bezel).

In another embodiment, multiple nodes may be configured to cover theentire I/O surface 52. By way of example, the multiple nodes may be laidout in a pixilated array of sensing nodes. For example, in the case ofmutual capacitance, the wires may be formed as a grid with a pluralityof sensing nodes.

In another embodiment, multiple nodes may be configured to cover somesub regions of the I/O surface 52. In one implementation, for example, amultiple nodes may be configured to cover all the areas that surroundthe various I/O devices 54. In another implementation, multiple nodesare configured to cover all the areas underneath the various I/O devices54. In another implementation, multiple nodes may be positioned at oneor more of the I/O devices 54 rather than the entire set of I/O devices54. These nodes may be positioned around and/or underneath the I/Odevices 54. In another implementation, multiple nodes may be positionedaround the edge of the I/O surface 52 (e.g., bezel).

The proximity based capacitive sensing circuit may be configured to worksimilarly to a touch based capacitive sensing circuit, and in some casesthe circuits may be one and the same. For example, using a smartcontroller, a touch based device may be configured to sense objects thatare not touching but in close proximity to the touch surface. Typically,the capacitance increases as a finger approaches the touch surface, andfurther increases as the finger presses on the touch surface.

FIG. 4 is a diagram of a method 80 of operating an I/O platform, inaccordance with one embodiment of the present invention. The methodbegins at block 82 where a proximity sensing field is generated above anI/O device. The sensing field may for example be generated usingtechniques associated with capacitance or infrared technologies (orothers mentioned above). The sensing field is spaced apart (non contact)but in close proximity to the I/O devices. The distance between the I/Odevice and the sensing field may be as small as about 2 mm and as largeas about 30 mm, and more particularly between about 2 mm and about 10mm. Capacitance sensing fields may for example be between about 2 mm andabout 3 mm while IR fields can be just about anywhere (e.g., focused toa particular distance above the surface).

In block 84, disturbances or changes in the sensing field are detectedand thereafter registered as a proximity event. For example, when afinger is placed within the sensing field, the sensing field isdisturbed and a controller translates this disturbance to mean that afinger is present. In the case of a capacitive proximity sensor, whichmay be separate component or a part of a capacitive sensing touch pad,the finger changes the capacitance at the sensing field and this changeis registered as a proximity event. In the case of an IR proximitysensor, the finger scatters the light from the emitter when it is placednear the sensing field, and the scattered light is measured by adetector. Changes in light intensity is registered as a proximity event.

Thereafter, in block 86, a proximity control signal is generated basedon the presence of the finger in the proximity sensing field. This istypically accomplished before any other touch based inputs are receivedsince the finger passes through the sensing field on its way toactuating the touch based I/O devices. The proximity control signal canbe used to affect the I/O platform in some non trivial manner or it canbe used as a control signal in a host device.

FIG. 5 is a simplified side elevation view of a portable computingdevice 100, in accordance with one embodiment of the present invention.The portable computing device 100 may for example be a portablecomputer, tablet PC, PDA, media player (e.g., music player, videoplayer, game player), digital camera, GPS module, and the like.

The portable computing device 100 may be a handheld computing device. Asused herein, the term “hand-held” means that the computing device has aform factor that is small enough to be comfortably held in one hand. Ahand-held computing device may be directed at one-handed operation ortwo-handed operation.

In one-handed operation, a single hand is used to both support thedevice as well as to perform operations with the user interface duringuse. Cellular phones, and media players are examples of hand-helddevices that can be operated solely with one hand. In the case of a cellphone, for example, a user may grasp the phone in one hand between thefingers and the palm and use the thumb to make entries using keys,buttons or a joy pad.

In two-handed operation, one hand is used to support the device whilethe other hand performs operations with a user interface during use oralternatively both hands support the device as well as performoperations during use. PDA's and game players are examples of hand-helddevice that are typically operated with two hands. In the case of thePDA, for example, the user may grasp the device with one hand and makeentries using the other hand, as for example using a stylus. In the caseof the game player, the user typically grasps the device in both handsand make entries using either or both hands while holding the device.

The hand-held device may further be sized for placement into a pocket ofthe user (e.g., pocket sized). By being pocket-sized, the user does nothave to directly carry the device and therefore the device can be takenalmost anywhere the user travels (e.g., the user is not limited bycarrying a large, bulky and often heavy device, as in a laptop ornotebook computer).

As shown, the computing device 100 generally includes a housing 102 forenclosing internally various electronic components associated withoperating the computing device 100. In general, the housing 102 servesto surround the internal electronic components at a peripheral regionthereof so as to cover and protect the internal components from adverseconditions. The internal electrical components may include mechanismsfor processing, generating, transmitting and receiving data associatedwith operating the portable computing device. By way of example, theinternal electronic components may take the form of integrated circuits(e.g., processors, controllers, memory), or other electrical devices(e.g., circuit boards, cables, fans, power supplies, batteries, harddrive, disk drive, modem, connectors, etc.).

The computing device 100 also includes a user interface 104 that is thepoint of communication between the device 100 and the user of the device100. The user interface may include any number of UI features, which canbe located on any of the surfaces of the housing 102. In most cases,however, a predominant number of the UI features are placed on a frontsurface of the computing device 100. The UI features 106 may be embodiedin a variety of ways and may include any number of input and/or outputdevices including but not limited to switches, buttons, dials, sliders,navigation pads, touch pads, touch screens, displays, speakers, and thelike.

The UI features may even include advanced devices such as mechanicallyactuated touch pads such as those described in U.S. patent applicationSer. No. 10/643,256, touch sensitive housings such as those described inU.S. patent application Ser. No. 11/115,539, multipoint sensing touchpads or touch screens such as those described in U.S. patent applicationSer. No. 10/840,862, display actuators such as those described in U.S.patent application Ser. No. 11/057,050, all of which are hereinincorporated by reference.

In one embodiment, the user interface 104 of the computing device 100includes a full screen or near full screen display 108. A full screendisplay is a display that consumes, or at least dominates, a surface(e.g., front surface) of the computing device 100. In one example, afull screen display consumes substantially the entire front surface ofthe computing device 100 (in both directions). The full screen displaymay for example consume at least 90% of a front surface of a housing forthe computing device. It may extend edge to edge or it may fit within asmall bezel of the housing at the edge of the device. The full screendisplay may have a variety of different configurations depending on theoverall footprint of the device. If the device is wide, the full screendisplay may have a traditional aspect ratio of about 4:3. If the deviceis elongated, the full screen display may have an aspect ratio that ismore panoramic such as 16:9.

In this embodiment, in order to provide input functionality, the userinterface 104 may additionally include a touch screen 110 disposed overthe full screen display 108. The touch screen 110 may extend completelyover the full screen or it may only cover a portion of the full screen.In most cases, the touch screen 110 covers at least the display portionof the full screen display 108. A multifunctional device that utilizes afull screen display among other advanced UI features may be found inU.S. Provisional Pat. App. No. 60/658,777, which is herein incorporatedby reference.

Although a touch screen display is shown, it should be appreciated thatthe user interface may include any combination of input output devicessuch as any of those described above, or any of those further describedin the applications incorporated by reference.

The computing device 100 also includes a proximity detection system 112.The proximity detection system is configured to detect when an objectsuch as a finger (or stylus) comes in close proximity to the computingdevice 100. Proximity sensing differs from touch sensing in that thereis no contact. The sensing occurs above the surface of the housing 102.For example, the proximity detection system 112 detects when an objectsuch as a finger is placed in space above, and not in contact with, asurface of one or more regions of the computing device 100. The regionsmay include any side or any portion of a side of the computing device100.

Proximity detection can be performed using a variety techniques. Thetechniques general depend on the object that is to be detected (e.g.,finger, plastic, metal), the environmental conditions surrounding thecomputing device, the size of the computing device, and the costsassociated with the proximity sensing technique, the resolution of theproximity sensing technique, etc. By way of example, the proximitydetect system 112 may be based on techniques such as capacitive,electric field, inductive, hall effect, reed, eddy current, magnetoresistive, optical shadow, optical visual light, optical IR, opticalcolor recognition, ultrasonic, acoustic emission, radar, sonar,conductive or resistive. In each of these cases, the proximity detectionsystem 112 may include one or more proximity sensors and/or switches.Switches are similar to sensors except that they provide a switch outputwhen an object is detected. Furthermore, depending on the technologyused, the proximity detect system may be hidden from view (e.g.,internal to the housing) or it may be in plane sight to the user (e.g.,external to the housing).

In one embodiment, the capacitance sensing circuit described in FIG. 3(or similar circuit) is used. In another embodiment, the IR sensingcircuit described in FIG. 2 (or similar circuit) is used.

In one embodiment, the proximity detection system 112 is configured todetect the presence of an object at the user interface 104 of thecomputing device 100. Using the illustrated example, the U.I. 104 islocated at the front surface of the housing 102, and thus the proximitydetection system 112 is configured to generate a sensing field 114 overall or some portion of the front surface of the housing 102. Asmentioned previously, this may be accomplished with single or multiplesensing nodes, and regionally or over the entire surface.

In one implementation, the user interface 104 is activated from adormant state when an object is detected. For example, the device may bewoken up from a sleep mode. When it wakes up, the display 108 may befully powered and the touch screen 110 may be readied for receivingtouch inputs (e.g., unlocked).

In another implementation, the features of the user interface arealtered when an object is detected. For example, buttons and switchesmay be illuminated and readied for input, or particular GUI elements ona display may brought into view and readied for receiving inputs. Thelater example may allow the display 108 to keep GUI controls and toolshidden until a user makes a motion to use the touch screen 110 or otherinput device. The GUI controls and tools that pop up when a finger isdetected may be standard set of tools or they may be based on otherfactors including the location of the finger and/or the applicationcurrently running in the computing device 100. For example, in a musicapplication, an onscreen control box that includes menu, play/pause,seek, and the like may be presented when a user hovers their fingerabove the user interface (especially the input portion of the userinterface). Alternatively, an onscreen or virtual scroll wheel may bepresented so that a user can scroll through a list of songs. Once theGUI control element is presented, the user may make selectionsassociated therewith by using one of the input devices provide by thecomputing device (e.g., touch screen) or by performing a proximitygesture. A proximity gesture can be similar to a touch gesture exceptthat it is performed without contact, and in some cases may even includea z component along with an x and y component.

FIG. 6 is a diagram of a proximity based method 150, in accordance withone embodiment of the present invention. The proximity based method maybe performed on any electronic device as for example any of thosepreviously described. By way of example, the method may be performed inthe computing device of FIG. 5. The method 150 includes at least twoblocks—blocks 152 and 154. In block 152, an object spaced away and inclose proximity to an electronic device is sensed (e.g., non contact).This may for example be accomplished with any of those proximitydetection systems described above. The sensing may be performed over theentire surface or regionally at particular locations of the electronicdevice as for example a single side, plurality of sides, all sides, aportion of single side, different portions of a single side, portions onmultiple sides, and the like. The sensing may be configured to detectthe presence of or lack of an object around the sensing field locatedabove and in close proximity to the surface of the electronic device.Alternatively or additionally, the sensing may be broken up into nodes,pixels or juxtaposed sensing areas that cooperate to monitor the motionof an object as it is moved in the sensing field (this may be analogousto tracking as for example when using a touch pad or touch screen).

In block 154, an action is performed in the electronic device when anobject is detected. The actions may be widely varied. The actions, mayfor example, include logging onto a computer or a computer network,loading a user profile associated with a user's preferred arrangement ofthe computer desktop, launching a particular program, opening a file ordocument, opening a particular GUI element, viewing a menu, making aselection, executing instructions, encrypting or decoding a message,operating an input/output device and/or the like.

FIG. 7 is a diagram of a method 200 of operating an electronic device,in accordance with one embodiment of the present invention. The methodbegins at block 202 where an object is sensed in the space above a userinterface. The user interface may include for example a display, touchpad or a touch screen display (or any of those mentioned above).

Following block 304, the method proceeds to block 204 where the userinterface or some portion of the user interface is activated or affectedin a non trivial manner when an object is detected above the userinterface. In one embodiment, the user interface is activated from adormant state as for example when the device is in a sleep mode to savebattery power or lock mode to prevent inadvertent entries at the userinterface. In another embodiment, individual features of the userinterface may be activated. For example, buttons, switches or navigationpads may be unlocked and illuminated. Alternatively or additionally, adisplay may display a GUI interface as for example one that includestools or controls for operating some aspect of the electronic device.

The method also includes block 206 where the activated portion of theuser interface is deactivated when an object is no longer detected. Thedeactivation may be instantaneous or it may be based on a time out. Inthe later case, the user interface may be deactivated when an object isnot detected for a preset amount of time (e.g., 1 minute).

FIG. 8 is a diagram of a proximity method 250 of operating an electronicdevice, in accordance with one embodiment of the present invention. Themethod begins at block 252 where the presence of an object is detectedover a user interface feature. The user interface feature may forexample be a display, touch screen display, touch pad or some othernavigation/selection based device.

Following block 252, the method proceeds to block 254 where a displayportion of the user interface displays and enables a particulargraphical user interface element when the presence of the object isdetected. By way of example, the graphical user interface element may bea tool box or control panel associated with the application currentlyrunning on the electronic device.

Following block 254, the method proceeds to block 256 where inputfeatures of the user interface are monitored for user inputs. Forexample, a touch screen, touch pad or some other navigation/selectionbased device may be used to generate user inputs. Alternatively oradditionally, a proximity based device may be used to generate userinputs.

Following block 256, the method proceeds to block 258 where one or moreactions are performed in the particular graphical user interface elementbased on the user inputs.

In block 260, the graphical user interface element is disabled andremoved from the display when the presence of the object is no longerdetected. This may be instantaneous or based on a time out.

In one embodiment, block 256 is associated with a touch sensing devicesuch as a touch screen or touch pad, and therefore block 256 includesmonitoring touch events that occur on the touch sensitive surface of thetouch sensing device. These events may include tracking and selectionevents, which are normal to touch sensing devices.

In one implementation, the touch sensing device is further configured todifferentiate between light touch interactions and hard touchinteractions that occur on the touch sensing surface. This may forexample be accomplished by monitoring the forces exerted on a touchsensing surface during a touch event. The light touch interactions canbe used to perform passive actions such as navigating through content,and content manipulation generally without causing a major event tooccur. Examples of passive events include moving a cursor, scrolling,panning, etc. The hard touch interactions, on the other hand, can beused to select on screen buttons or initiate commands (e.g., causes asignificant change to occur). In this embodiment, the inputs may includethree input levels, proximity, light touch and hard touch.

FIG. 9 is a diagram of a touch method 300, in accordance with oneembodiment of the present invention. The touch method 300 may forexample be included in block 256 of FIG. 8. The touch method 300 beginsat block 302 where one or more touches are detected. The touches includenot only x any y components but also z components. The x and ycomponents are supplied by a touch sensing device such as touch screen,touch pad or touch housing. The z component, on the other hand, isgenerally provided by force sensors located behind the touch surface ofthe touch sensing device. By way of example, the x, y and z componentsmay be supplied by a display arrangement that includes a touch screenpositioned over a display and a force sensing device disposed underneaththe display. The x, y and z components may also be supplied by a touchpad or touch housing that includes a touch sensitive surface and forcesensing device underneath the touch sensitive surface. Additionally oralternatively, the z component may be supplied by a physical switchlocated behind the touch surface (display actuator), or possibly usingthe touch surface itself (e.g., contact patch areas that increase insize based on increases in touch pressure).

Following block 302, the method proceeds to block 304 where adetermination is made as to whether the touch is a light or hard touch(e.g., differentiating whether the detected touch is a light touch or ahard touch). The determination is generally based on the force orpressure of the touch (z component). For example, if the force of thetouch is smaller than a predetermined threshold then the touch isconsidered a light touch and if the force of the touch is larger thanthe predetermined threshold then the touch is considered a hard touch.

If it is determined that the touch is a light touch, the method proceedsto block 306 where a passive action is initiated when a touch event isperformed. Examples of passive actions include navigation, moving acursor, scrolling, panning, etc. If it is determined that the touch ishard touch, an active action is initiated when a touch event isperformed. Examples of active actions include making selections andimplementing commands.

Although not a requirement, the touch method may additionally include ablock where the one or more touches are classified as a primary touch ora secondary touch. Primary touches are touches that are intended tocause an action while secondary touches are touches that are notintended to cause an action. Gestures are examples of primary toucheswhile a thumb positioned over the touch area in order to hold thehand-held device is an example of a secondary touch. Once the touchesare classified as primary or secondary, the secondary touches arefiltered out, and the determination of whether a touch is a light orhard touch is made with the primary touches.

FIG. 10 is user interface method 400, in accordance with one embodimentof the present invention. The user interface method 400 may for examplebe performed on a computing device having a display, a touch sensitiveinput device such as a touch screen and a proximity detector. The userinterface method 400 begins at block 402 where an object such as afinger or stylus is detected above and spaced away from the display.This may be accomplished with the proximity detector when the fingercrosses into the sensing field of the proximity detector. In some cases,the touch screen may act as a proximity detector as for example in caseswhere the touch screen is based on capacitance. The controller can beconfigured to monitor changes in capacitance even when there is notouch. The object however typically has to be fairly close to the touchsurface of the touch screen.

Once the object is detected, the user interface method 400 proceeds toblock 404 where a user interface (UI) mode is determined. The userinterface mode may be widely varied. The user interface mode may includescroll modes, data entry modes, edit modes, control modes, informationmodes, etc. Each mode typically has one or more GUI interface elementsassociated therewith. By way of example, a virtual scroll wheel orslider bar may be associated with a scroll mode, a keyboard or keypadmay be associated with data entry mode, a tool bar such as a formattingtool bar or drawing tool bar may be associated with an edit mode, acontrol panel including buttons may be associated with a control mode, awindow may be associated with an information mode, etc.

The user interface mode may be determined at block 404 based on one ormore conditions including for example, one or more applicationscurrently running on the computing device, the current state or mode ofthe one or more applications and/or the proximity characteristics (e.g.,location). In fact, determining the user interface mode at block 404 mayinvolve monitoring and analyzing one or more conditions.

The current applications may for example include operating systems(e.g., Mac OS), word processing programs, spreadsheet programs, drawediting programs, image editing programs, gaming programs, photomanagement programs (e.g., iPhoto), music management programs (e.g.,iTunes), video editing programs (e.g., iMovie), movie managementprograms (e.g., QuickTime), music editing programs (e.g., GarageBand),Internet interface programs and/or the like.

The current state or mode of the applications may correspond to anactive portion of the application (e.g., current window or windowswithin windows). For example, the active portion of a music managementprogram may correspond to a music control mode, a playlist select mode,a menu mode, and/or the like. Further, the active portion of a photomanagement program may correspond to photo browsing mode or photoediting mode. Further still, the active portion of an internet interfaceprogram may correspond to a web mode or an email mode.

The proximity characteristics on the other hand may, for example,correspond to location, number of detected objects, etc.

With regards to applications, different applications may indicatedifferent UI modes. For example, a word processing or spreadsheetapplication may indicate a data entry mode, while a music managementprogram may indicate a control or scrolling mode. With regards to thecurrent state of an application, different modes of the application mayindicate different UI modes. For example, in a music management program,a menu window may indicate one UI mode, while a play list window mayindicate another UI mode.

With regards to proximity, the number of fingers may indicate differentUI modes. For example, one finger may indicate a first mode while twofingers may indicate a second mode. In addition, the location of thefinger(s) may indicate different UI modes. For example, a first locationmay indicate a first UI mode, while a second location may indicate asecond UI mode (if the object is located over a border of a musicprogram a first UI mode may be implemented, and if the object is locatedover a play list or list of songs in the music program a second UI modemay be implemented.

In one embodiment, the user interface mode is based on only one of theconditions. For example, the user interface mode is only based on theapplication, the current state of the application or one of the variousproximity characteristics as described above. In another embodiment, theuser interface mode is based on multiple conditions. For example, theuser interface mode may be based on a combination of at least twoselected from the application, the current state of the application andthe various proximity characteristics. By way of example, theapplication combined with a first proximity characteristic may indicatea first UI mode and the same application combined with a secondproximity characteristic may indicate a second UI mode.

To cite a few examples, if the application is a word processing orspreadsheet program then the mode may be determined to be a data entrymode so that data can be entered into the spreadsheet (e.g., keyboard).If the application is a music management program and a playlist iscurrently showing (active portion), the mode may be determined to be ascroll mode so that the items in the list may be scrolled through inorder to find a desired item (e.g., scroll wheel). Alternatively, if asong is playing (active portion), the mode may be determined to be acontrol mode so that the manner in which songs are played can becontrolled (e.g., play, stop, seek and volume control options). Further,if the application is a photo management program and a particular photois displayed (active portion), the mode may be determined to be acontrol mode so that the photo can be modified (e.g., converting toblack and white, removing red eye, and rotate options).

After determining the user interface mode 404, the user interface method400 proceeds to block 406 where one or more GUI elements are displayedbased on the user interface mode. This may be accomplished with thedisplay device. In some cases, only one GUI element is displayed and inother cases multiple GUI elements are displayed. The GUI element istypically associated with a particular mode. For example, a slider baror scroll wheel may be displayed in a scroll mode, a keyboard or keypadmay be displayed in a data entry mode, a tool bar may be displayed in anedit mode, various buttons or a control panel may be displayed in acontrol mode, and information windows may be displayed in an informationmode.

The GUI element may be displayed in a variety of ways. For example, itcan be positioned over the currently displayed graphical images, or itcan displace the currently displayed graphical images (e.g., minimize,shift, etc.). In some cases, the GUI element is made semi transparent sothat the current graphical image disposed below the GUI element can beseen (thereby eliminating minimizing and shifting). This may be helpfulwhen using a scroll wheel to traverse through a list disposed below thescroll wheel. Furthermore, the GUI element can be placed in the vicinityof the touch or it can be placed in some predetermined location. Thepredetermined location may be based on ergonomics, i.e., what is thebest location for the user.

In addition to the above, the GUI element may be displayed using atransition effect such as growing, fading in, popping up, and in somecases may even pulsate, throb, etc. If the effect is popping up, the GUIelement is immediately brought into view. If the effect is growing, asmall GUI element (e.g., scroll wheel) is initially displayed, andthereafter the GUI element continuously enlarges through various sizesuntil it reaches its desired size. In addition, the final size of theGUI element may be based on the length of time the object is detected.For example, the GUI element stops growing when the touch is no longerdetected. Alternatively, the speed and size may be user adjustable asfor example through a control panel. If the effect is fading, the GUIelement is slowly brought into view from nothing, through various levelsof distortion or transparency, to a final complete image. The fading canbe controlled similarly to growth. For example, the level of fade may becontrolled by the length of time the object is detected.

The transition effect may even carry over to the currently displayedimages, i.e., the images currently displayed before the object wasdetected. In one embodiment, the opposite effect happens to thecurrently displayed images. For example, the currently displayedgraphical images are minimized smaller and smaller as the GUI elementgrows larger and larger. Alternatively, if the GUI element immediatelypops in, the currently displayed graphical images can immediately popout or be immediately minimized.

In block 408, the functionality of the GUI element is enabled. Whenenabled, the GUI element is readied for receiving inputs. In the case ofa touch screen, for example, touch events are monitored relative to theGUI element and actions associated with the touch event are performed.The enablement may occur simultaneously with the display of the GUIelement so that a user can immediately start using the GUI element oncedisplayed. By way of example, in a scroll mode, a virtual scroll wheelmay be displayed and when enabled, the touch events are monitoredrelative to the scroll wheel. During monitoring, control signalsassociated with the position of the finger on the scroll wheel aregenerated as the finger swirls around the virtual scroll wheel. Thesesignals can be used to perform scrolling. For example, the number,combination and frequency of the signals may be converted into distance,direction and speed necessary for moving a selection bar through a list.Examples of virtual scroll wheels and their operations may be found inU.S. patent application Ser. Nos. 10/903,964 and 11/038,590, both ofwhich are herein incorporated by reference.

At some point after enabling and displaying the GUI element, adetermination 412 is made as to whether or not to deactivate the GUIelement. The determination 412 can be made in a variety of waysincluding, for example: 1) the object is no longer detected, 2) anobject has not been detected for a preset amount of time, 3) a time outoccurs (a preset amount of time has gone by since the GUI element wasfirst displayed/enabled), or 4) a user selection (e.g., a user selects abutton that closes the GUI element).

If the determination indicates deactivation, then the method proceeds toblock 414 where the GUI element is disabled and removed from display.Once disabled, actions will no longer be performed when a touch eventoccurs. The removal of the GUI element from display may function similarto displaying the GUI element in that it can be removed using atransition effect such as slowly fading out, shrinking or immediatelydisappearing (popping out). The removal transition effect may workopposite the displaying transitioning effect. For example, the GUIelement fades out similarly to fading in, shrinks similarly to growth orpops out similarly to popping in. Further, the GUI element can slowlyrecess and disappear from view while the displaced or shrunken currentgraphical images can slowly grow back to their original size and shape.If the determination does not indicate deactivation, then the methodmaintains the display of the GUI element as well as the enablementthereof.

FIG. 11 is a determination method 450 in accordance with one embodimentof the present invention. The determination method may, for example,correspond to the block 404 in FIG. 10. The determination method beginsat block 452 where the current application is determined. Thedetermination method proceeds to block 454 where the current state ofthe application is determined. Following block 454, the determinationmethod proceeds to block 456 where the proximity characteristicsassociated are determined. The determination method proceeds to block460 where a UI mode is selected based on the results from blocks452-458. By way of example, a set of rules may indicate appropriate UImodes for a particular set of conditions.

FIG. 12 is a user interface method 500 in accordance with one embodimentof the present invention. The method may, for example, be performed on acomputing device having a display, a touch sensitive input device suchas a touch screen and a proximity detector. The interface method 500begins at block 502 where a list of songs are displayed. FIG. 13A showsone example of a window 530A including a list of songs 532A, and FIG.14A shows another example of a window 530B including a list of songs532B. FIG. 13A may, for example, be a graphical user interface displayedon an iPod® manufactured by Apple Computer of Cupertino, Calif., andFIG. 14A may for example be a graphical user interface associated with amusic management program, such as iTunes® manufactured by Apple Computerof Cupertino, Calif.

Following block 502, the user interface method 500 proceeds to block 504where an object is detected over the displayed list of songs (or windowor entire GUI). This may be accomplished with the proximity detectorwhen an object such as a stylus or one or more fingers is placed abovethe touch screen display. FIGS. 13B and 14B show a finger 525 placedover the window 530 including the list of songs 532. Although shown inthis figure, the finger is spaced away and not in contact with the touchsurface.

Once the presence of an object is detected, the user interface method500 proceeds to block 506 where a virtual scroll wheel is activated.That is, a virtual scroll wheel is displayed in addition to the list ofsongs and its functionality is enabled. In essence, because the songlist was active, a scroll wheel that allows a user to traverse throughthe songs in the list of songs is provided. In some cases, the virtualscroll wheel displaces the media items, i.e., the media items areminimized or shifted to make room for the virtual scroll wheel. In othercases, the virtual scroll wheel is positioned or laid over the mediaitems (the media items keep their current size, shape and position). Thevirtual scroll wheel can be made semi-transparent so that the mediaitems can be viewed through the virtual scroll wheel. FIGS. 13C and 14Cshow a transparent virtual scroll 936 wheel laid over the window 530including the list of songs 532. Alternatively, a virtual slider bar maybe displayed.

Once displayed, a determination 508 is made as to whether or not ascrolling event (or gesture) is performed relative to the virtual scrollwheel. For example, whether or not a finger is positioned over thescroll wheel and whether or not the finger is moved around the scrollwheel in a swirling fashion. The scroll event may be a proximity eventor a touch event.

If a scrolling event is performed by the user, the user interface method500 proceeds to block 510 where scrolling is implemented through thelist of songs in accordance with the scrolling touch event. By way ofexample, a selector bar may be moved from one song to another as thefinger is swirled around the virtual scroll wheel. FIGS. 13D and 14Dshow the finger 525 swirling around the virtual scroll wheel 536, and aselector bar 538 moving linearly through the list of songs 532 inaccordance with the swirling finger 525. In the illustrated embodiments,the selector bar is moved linearly up when the finger is swirled in aclockwise fashion and linearly down when the finger is swirled in acounterclockwise fashion. It should be noted, however, that this is nota limitation. For example, the selector bar may moved linearly down whenthe finger is swirled in a clockwise fashion and linearly up when thefinger is swirled in a counterclockwise fashion.

If a scrolling or select touch event is not performed, the userinterface method 500 proceeds to block 516 where the virtual scrollwheel is deactivated. That is, the virtual scroll wheel is disabled andremoved from the display. FIGS. 13E and 145E show the display 528without the virtual scroll wheel 536. Although the virtual scroll wheel536 is removed, changes made to the list of songs, i.e., the position ofthe selector bar 538, typically remain.

In some cases, the virtual scroll wheel may include button zones acrossits surface or a virtual button at its center or around its sides. Thebuttons and button zones may for example correspond to menu, play, seek,pause, and/or the like. In this particular embodiment, the methoddescribed above may include additional steps that occur before block516. For example, if a scrolling touch event is not performed, the userinterface method 500 may include an additional block where adetermination is made as to whether or not a selection touch event (orgesture) is performed relative to the virtual scroll wheel. Theselection touch event may be implemented by tapping the button or byexerting increased or decreased pressure on the button rather thanswirling around the surface of the virtual scroll wheel (see FIGS. 13Fand 14F). If the button is a song select or enter button, the methodinclude another block where the song with the selector bar disposedthereover is selected. That is, when the virtual button is tapped, orotherwise selected, the song currently covered by the selector bar isplayed and outputted for the user's enjoyment.

It should be noted that the methods described above are not limited toscrolling through a list of songs. Any media item as well as any groupof elements can be scrolled through using the aforementioned technique.For example, in photo layout 542 as shown in FIGS. 15A-15C, the virtualscroll wheel 536 may appear when the user places their finger 525 overthe photo layout 542 (or grouping), and thereafter it can be used tomove a highlighter 544 through the various photos 543 in the layout 542.By way of example, the photos may be thumbnails images that maketraversing through a large number of images easier.

FIG. 16 is a method 600, in accordance with one embodiment of thepresent invention. The method begins at block 602 where it is determinedif an object in space is detected. If an object is detected, the methodproceeds to block 604 where the current operating conditions aremonitored and analyzed. The conditions may for example correspond to thecurrent application, the state of the application and/or the proximitycharacteristics associated with the object.

If a first set of conditions are implemented, the method proceeds toblock 606 where a first GUI element is activated. For example, as shownin FIGS. 17A-B, in an active window 660 of a music management program, ascroll wheel 662 may be activated when a user places their finger abovea playlist portion 664 of the active window 660 without touching thetouch surface.

If a second set of conditions are implemented, the method proceeds toblock 608 where a second GUI element is activated. For example, as shownin FIGS. 17B-C, in the active window 660 of a music management program,a music control panel 667 may be activated when a user also touches aborder 668 of the active window 660. Although they work independent ofone another, the first and second GUI elements may be activated at thesame time if the first and second conditions occur simultaneously (FIG.13C).

Following block 606, the method proceeds to block 610 where it isdetermined if the first GUI element should be deactivated. If so, themethod proceeds to block 612 where the GUI element is deactivated. Forexample, as shown in FIG. 17D, the first GUI element (scroll wheel 662)is disabled and removed from display when the finger 625 is no longerdetected over the playlist 662. If not, the method maintains block 606.

Similarly but independently, following block 608, the method proceeds toblock 614 where it is determined if the second GUI element should bedeactivated. If so, the method proceeds to block 616 where the GUIelement is deactivated. For example, as shown in FIG. 17E, the secondGUI element (control panel 667) is disabled and removed from displaywhen the finger 625 is no longer detected over the border 668. If not,the method maintains block 608.

It should be noted that the method is not limited to only two GUIelements and that other GUI elements may be activated if otherconditions are implemented (N+1). For example, as shown in FIG. 17F, theuser may move their finger 625 from over the border 668 to over a menuportion 670 of the active window 660 thereby initiating a change fromthe control panel 667 to a scroll wheel 672 (e.g., while the second GUIelement is being deactivated, the third GUI element is being activated).

Further, as shown in FIG. 17G, the user may add another finger 625 tothe mix thereby initiating a change from the first control panel 667 toa second control panel 682. The first control panel 667 may include afirst set of control options such as play, stop, seek and volume optionsand the second control panel 682 may include a second set of controloptions such as song playing order, song information, light effectoptions.

Moreover, as shown in FIG. 17H, the user may place one finger 625A overthe border 668, another finger 625B over a menu portion 670, and yetanother finger 625C over the playlist portion 664 thereby initiatingthree different GUI elements, particularly, a control panel 667, a firstscroll wheel 672 for scrolling through the menu 670, and a second scrollwheel 662 for scrolling through the playlist 664.

In addition, multiple GUI elements can be activated in the same portion.For example, as shown in FIGS. 171 and 17J, if the user places theirfinger over a particular box 690 in the playlist 664, a keyboard 692 maybe activated so that the user can enter data associated with the song(e.g., title, artist, genre, etc.). If the scroll wheel 662 is active atthe same time as the keyboard 692, the scroll wheel 662 may be minimizedto accommodate the keyboard 692 as shown. Once the keyboard 992 isdeactivated, the scroll wheel 662 reverts back to its original size.

FIG. 18 is a diagram of a GUI operational method 700, in accordance withone embodiment of the present invention. The GUI operational method 700is configured for initiating zooming targets. The GUI operational method700 generally begins at block 702 where a control box GUI element isdisplayed. The control box contains one or more control buttons, whichare somewhat close together, and which can be used to perform actions.The control box may, for example, include control buttons such asmaximize, minimize, close, and the like. Following block 702, the GUIoperational method 700 proceeds to block 704 where the control box isenlarged, or at least one of the control buttons is enlarged for aperiod of time when the presence of an object is detected above thetouch screen and over the control box or one of the control buttons. Inthe case where the control box is enlarged each of the control buttonsis enlarged thereby making selection thereof much easier. In the casewhere only the control button is enlarged, the user would decide whetherthis is the correct button and if so select the enlarged control button,or restart the process so that the appropriate control button ispresented. In most cases, the size of the control buttons corresponds tothe size of the finger so that they may be easily selected by theobject. Following block 704, the GUI operational method 700 proceeds toblock 706 where a control signal associated with the selected controlbutton is generated when the object is placed on the touch screen andover one of the enlarged control buttons.

FIGS. 19A-19D illustrate a zooming target sequence using the GUIoperational method 700 described above. As shown in FIG. 19A, a user 710places their finger 776 above the touch screen and over a control box778. Because the buttons 780 of the control box 778 included therein aresmaller than the finger 776 and located close together, it is difficultfor the user 710 to make a selection directly without possibly pressingan undesirable button 780, e.g., a button adjacent the desired button.By way of example, the finger 776 may cover two or more of the buttons780. As shown in FIG. 19B, at least a portion of the control box 778 isenlarged including the buttons 780 included therein when the finger isdetected. As shown in FIG. 19C, once the control box has reached itsenlarged state, the user can select one of the enlarged buttons, whichis now closer to the size of the finger using the touch screen. By wayof example, the user may tap on the desired control button. As shown inFIG. 19D, the control box reduces to its initial size after the buttonis selected or after a predetermined time period in which no selectionwas made (e.g., times out) or when the user moves their fingersubstantially away from touch screen (out of the proximity sensing area)or when the user moves their finger out of the area of the control boxwhile still in the sensing area of the proximity detector.

FIG. 20 is an expansion method 800, in accordance with one embodiment ofthe present invention. The method may be implemented on computingdevices having a touch screen display and a proximity detector. Themethod generally begins at block 802 where graphical information ispresented on the touchscreen display. This is typically accomplishedwith the display portion of the touchscreen display. In most cases thegraphical information fills the entire display screen, although in somecases it may only fill a portion of the display screen. The graphicalinformation may for example be a graphical user interface (GUI). As isgenerally well known, the GUI represents, programs, files andoperational options with graphical images. The graphical images mayinclude windows, fields, dialog boxes, menus, icons, buttons, cursors,scroll bars, etc. During operation, the user can select and activatevarious graphical images in order to initiate functions and tasksassociated therewith. By way of example, a user may select a button thatopens, closes, minimizes, or maximizes a window, an icon that launches aparticular program or a link that opens a particular web page. Inaddition, the user may actuate user interface controls such as a scrollbar to perform scrolling in the GUI. The GUI can additionally oralternatively display non interactive text and graphics

Following block 802, the method proceeds to block 804 where an object isdetected over the touchscreen display. This is generally accomplishedwith the proximity detector and/or the touchscreen display (e.g.,capacitive). Depending on the configuration of the proximity detector,the proximity detector may recognizes one or more objects, as well asthe position and magnitude of objects in the sensing region of theproximity detector.

Following block 804, the method proceeds to block 806 where an area orportion of the touchscreen display is expanded proximate the location ofthe object. Any portion of the touchscreen display can be expanded. Thestep of expanding may include visually expanding a portion of thepresented graphical information in the region of the object relative tothe remaining portions of the presented graphical information outsidethe region of the object. The step of expanding may also includeexpanding the touch sensitive portions associated with the expandedvisual portion. The sensitive portion are points or regions of thetouchscreen that are linked to particular points or regions of thegraphical information as for example, the touch sensitive region above agraphical button. In most cases, the sensitive portions are scaled withthe visually expanded portions. That is, the step of expanding includesscaling the input sensitivity with the expanded visual portions of thegraphical information. The scaling may be proportional since thesensitive portions are typically a little larger than the visualportions (the visual portion and the sensitive portion aren't exactly a1:1 relationship). By way of example, if the visual portion grows by afactor of 3 then the sensitive portion grows by a factor of 3.

During expansion, the visual and/or sensitive portions of thetouchscreen display may be magnified and/or increased in size comparedto the remaining visual and/or sensitive portions of the touchscreendisplay. By way of example, the visual portion may grow from itsoriginal size to an enlarged size (e.g., from 4 mm to 8 mm) and bemagnified from its original state to a magnified state (e.g., from 1× to2×). Although the expanded portions are enlarged and magnified, in mostcases, the expanded portion maintains the same aspect ratio and shapethat it has in the unexpended state. As should be appreciated, expansionmay be somewhat analogous to zooming.

The area that is expanded may be the area underneath the object.Alternatively, the area that is expanded may be offset from the objectsuch as for example above, below or to the sides of the object. This mayallow the user to more clearly view the expanded area.

The size, magnification and shape of the expanded area may be widelyvaried. By way of example, and not by way of limitation, the expandedarea may have an area of between about 100 mm² and about 400 mm², and aheight and width between about 10 mm to about 20 mm. Furthermore, theexpanded portions may be magnified between greater than 1 time to over athousand times, more particularly between about 2× to about 100×, andeven more particularly, the between about 2× and about 20×. Moreover,the expanded area may be formed from various shapes including but notlimited to circles, ovals, squares, rectangles, triangles, and the like.Other shapes such as symbols, logos, characters may also be used.

In one embodiment, the expanded portion of the graphical information israised relative to the non expanded portion of the graphicalinformation. For example, the expanded portion may appear to protrudeaway from the non expanded portions. This is typically done graphicallyas both the expanded and non expanded portions are typically produced inthe same plane. By way of example, shading principals may be used tomake the expanded portion appear to protrude away from the non expandedportions.

In some cases, the expanded portion includes a substantially planarplateau region and a substantially sloped transition region. The planarplateau region provides a planar surface for displaying the targetedgraphical information and the transition region provides a gradient ofgrowth between the plateau region and the non expanded portion of thegraphical information. The transition region, which is the edge of theplateau compacts or compresses the graphical information located betweenthe plateau and the non expanded portion of the graphical information.For example, the graphical information contained in the transitionregion have compacted or compressed magnification levels (this maydistort the graphical information contained therein) In alternativeimplementations, the plateau may be rounded rather than being planar orthe expanded portion may be one large transition region without having aplateau. In either case, the expanded portion may look like a roundedpimple or bump.

The expanded portion may be a localized area, which can be any portionof the graphical information. The localized area may include any portionof the graphical information including but not limited to background(e.g., wall paper), windows, fields, text, dialog boxes, menus, icons,buttons, cursors, UI controls or any combination thereof.

The expanded portion may also be linked to a particular object of thegraphical information. For example, a particular window, field, dialogbox, menu, icon, button, tool bar, user interface element (e.g., scrollbar, scroll wheel, slider bar, dial), control box, footnote and thelike. In some case, the entire object is expanded. For example, when thefinger is placed over a window, the entire window is expanded. In othercases, only a portion of the object is expanded. For example, when thefinger is placed over a tool bar, only the selectable items areexpanded. As should be appreciated, these objects may need to beexpanded so that they can be easily used by a human hand.

The time when expansion takes place can be widely varied. In oneembodiment, expansion is activated immediately after the touch isdetected. In another embodiment, expansion is activated after the objectis detected for a predetermined amount of time. In cases such as this,the user may have to hover their finger over the area desired to beexpanded for the predetermined amount of time in order to initiate theexpansion. By way of example, the dwell time may be between about 0.5 toabout 5 seconds, and more particularly about 1 second. This embodimentmay be employed to prevent inadvertent expansion. That is, the timedelay may be used to avoid implementing expansion with casual contactnot intended for expansion.

Once expansion is activated, the speed or rate at which the expandedarea expands may be widely varied. The growth can happen quickly orslowly. In one embodiment, the expanded area grows from its normal stateto the expanded state almost instantaneously. In another embodiment, theexpanded area grows over some predetermined amount of time, i.e., thearea expands gradually over time. The rate may be preset or it may bebased on some external factor. For example, the rate of growth may bebased on the touch pressure, i.e., the greater the touch pressure, thegreater the rate of change. Alternatively, the rate of growth may bebased on multiple taps, i.e., each tap causes incremental expansion.

The manner in which expansion is implemented may also be widely varied.In one embodiment, the amount of expansion (size, magnification, etc) ispreset. The preset amount may be fixed or it may be adjustable. Forexample, a user may adjust the characteristics of expansion via acontrol menu. In another embodiment, the amount of expansion is based onsome external factor. In one case, the amount of expansion may be basedon the location of the object relative to the graphical information(e.g., region sensitivity). For example, a first located object of thegraphical information may be magnified 2× and a second located object ofthe graphical information may be magnified 16×. In yet anotherembodiment, the expanded portion may be configured to follow the objectas the object is moved across and above the touchscreen display. Forexample, the location of the expanded portion changes in accordance withthe location of the object, i.e., it mirrors the position of the object.

Although not shown in FIG. 20, the method may include additional stepssuch as reverting back to a non expanded state when the object is nolonger detected. In one embodiment, the expansion is deactivatedimmediately after the object is no longer detected thereby causing theexpanded portion to revert back to its normal state. In anotherembodiment, expansion is deactivated after a predetermined amount oftime, i.e., there is a lag time. The lag time may be preset (e.g., usersettable) or it may be based on external factors such as the amount ofinformation located in the expanded area and human factors such as howlong the user would take to read or grasp this particular amount ofinformation. The lag time may allow the user to establish if theexpanded area is the desired target. If its not the desired target, theuser can move their finger to a new target. If it is the desired target,the user may perform additionally steps within the expanded area. By wayof example, and not by way of limitation the lag time may be betweenabout 0.5 to about 5 seconds, and more particularly 1 second.

The speed or rate at which the expanded area reverts back to its normalstate may be widely varied. The atrophy can happen quickly or slowly. Inone embodiment, the expanded area atrophies from its expanded state tothe normal state almost instantaneously. In another embodiment, theexpanded area atrophies over some predetermined amount of time, i.e.,the area atrophies gradually over time.

The method may also include receiving inputs within the expanded area.The inputs may for example be touch events such as a selection inputthat is implemented with tapping or increased touch pressure.Alternatively, the input may be a gestural input or a data entry input.By way of example, in an expanded state, a virtual scroll wheel may becapable of being manipulated by a swirling finger gesture or text may beentered into an expanded document. In all of these cases, the expansionmay be designed to expand a feature or group of features (such asbuttons) so that they are adequately sized for finger manipulation.

Moreover, the method may include detecting a second object over thetouchscreen display and expanding a second area of the touchscreendisplay proximate to the location of the second object. In oneimplementation, the second object is detected at the same time as thefirst object such that the first and second expanded areas are expandedsimultaneously (as for example using a multipoint proximity detectionsystem). In another implementation, the second object is detected afterthe completion of the first object. In cases such as this, if theplacement of the second object occurs during the lag time of the firstexpansion, then the first expansion may smoothly transition to thesecond expansion (e.g., as the first gradually shrinks, the secondgradually expands). Alternatively, a second object in the region of theexpanded area may cause further expansion. For example, if the expandedportion includes a plurality of features, a second object may be used tofurther expand one or more of the features. Thereafter, the feature maybe selected for example by performing a touch event on the touch screen.

Alternatively, expansion may not be implemented if more than one objectis detected at the same time, i.e., simultaneously. This “double object”could be considered a null input for the purposes of expansion.

The methods mentioned above may be implemented with software orhardware, or a combination of hardware and software. In one embodiment,the method is implemented by an operating system. As such, the methodmay be implemented during any program or application running inconjunction with the operating system. That is, expansion can occur inany program or application. By way of example, the operating system maycorrespond to Mac OS, OS/2, DOS, Unix, Linux, Palm OS, and the like. Theoperating system can also be a special purpose operating system, such asmay be used for limited purpose appliance-type computing devices.Operating systems are generally well known and will not be described ingreater detail.

FIG. 21 is a proximity method 850, in accordance with one embodiment ofthe present invention. The method begins at block 852 where objectsensing is performed via a proximity detection system. The object mayfor example be a finger or stylus. The object is sensed when the objectis placed over, but not in contact with a touchscreen display.

If an object is sensed, the method proceeds to block 854 where the areanear the sensed object is expanded. In particular, a portion of the GUIin the vicinity of the object is visually expanded relative to theremaining portions of the GUI. The touch sensitivity associated with theGUI is also expanded. The expanded area may be expanded immediately orit may appear to gradually grow until it reaches its final shape size,elevation, etc.

In most cases, the visually expanded portion is made to appear like itis protruding out of the image plane. The protruding effect is typicallyaccomplished through graphics such as shading or other similar means.The protruding effect may be implemented in a localized area of the GUIor it may be implemented at a particular GUI image such as a controlbox, tool bar, user interface element, and/or the like.

In one embodiment, the expanded portion includes a plateau region and atransition region. During expansion, the target area contained withinthe plateau region increases in size and magnification. At the sametime, the transition region compresses the GUI between the target areacontained in the plateau region and the remaining unexpanded portions ofthe GUI. As a result, the plateau region appears to raise above theremaining portions of the GUI.

Following block 854, the method proceeds to block 856 where adetermination is made as to whether or not features are located withinthe expanded area. The features may be buttons, web links, icons, userinterface elements, data entry regions and/or the like.

If a feature is located within the expanded area, the method proceeds toblock 858 where a determination is made as to whether or not the featureis actuated via a touch event (e.g., selected, activated, etc.). Forexample, the object changes from hovering over the touch surface of thetouchscreen to actually touching the touchscreen. The determination mayinclude monitoring touches that occur after a hover and associating orlinking a particular feature to the touching object. The association orlink may be made by ranking the relevance of the touching object to thefeature. The ranking may for example include producing a figure ofmerit, and whichever feature has the highest figure of merit, giving itsole access to the events occurring with the touching object. By way ofexample, the ranking may include calculating the centroid of a touch andits proximity to the feature.

If the feature is a user interface element, the feature may be actuatedvia a touch gesture. See for example, U.S. patent application Ser. No.10/903,964, which is herein incorporated by reference. If the feature isa data entry feature, the feature may be actuated by entering data asfor example through the use of a cursor or keyboard (e.g., wordprocessing).

In cases where the expanded area includes a plurality of features, anintermediate block may be included between blocks 856 and 858. Theintermediate block includes providing feedback as to which feature isthe most likely selectable feature based on some predefined criteria(the feedback indicates where a system thinks the user is targeting).The criteria may for example be based on the location of the spaced awayobject relative the location of the various features. For example, thefeature located closest to the object may indicate that it is the mostlikely candidate for selection. In essence, the feedback is aconfirmation of correct targeting before selection. If the user does notget the right feedback, the user has the option of moving their fingerto another location. The feedback may be visual or auditory. By way ofexample, visual feedback may include modifying the feature such aschanging its color, position, size, shape, symbol, etc., and auditoryindication may include providing sounds associated with the feature.

Once the feature is actuated, the method proceeds to block 860 where anaction associated with the feature is performed. The action may forexample include launching a particular program, opening a file ordocument, viewing a menu, making a selection, executing instructions,generating control signals, and the like.

If the feature is not actuated or if there is no feature located withinthe expanded area, the method proceeds to block 862 where adetermination is made as to whether or not the object (finger) is movingin the sensing field of the proximity detector. For example, if theobject is stationary in the sensing field or whether the object is beingmoved about the sending field.

If the object is moving, the method proceeds to block 864 where theexpanded area is moved in accordance with the location of the movingobject. That is, the expanded area follows the motion of the object.

If it is determined that the object is not moving, the method proceedsto block 866 where a determination is made as whether or not the objectis still sensed. If so, the method proceeds to block 868 where thepresent expanded area is maintained. Thereafter, the method proceedsback to block 856. If the object is not sensed, the method proceeds toblock 870 where the expanded area is reduced to its normal state. Thereduction may mimic the expansion, but in a reverse direction. By way ofexample, in embodiments where the expanded portion includes a plateauregion and a transition region, the target area contained within theplateau region may be decreased in size and magnification. At the sametime, the transition region may decompress the GUI between the targetarea contained in the plateau region and the remaining unexpandedportions of the GUI. As a result, the plateau region appears tointegrate back into the remaining portions of the GUI. Furthermore, thereduction may occur immediately after the object is not sensed or aftersome predetermined amount of time.

FIG. 22 is a proximity gesture method 900, in accordance with oneembodiment of the present invention. The method generally starts atblock 902 where a proximity sensing field is provided. The proximityfield may be a single node proximity field or a pixilated array ofsensing nodes.

The method proceeds to block 904 where disturbances or changes in thesensing field are monitored. For example, the time between disturbances(tap) and/or the time of the disturbances (lag) and/or the location ofdistinct disturbances (first tap, second tap), varying locations of asingle disturbance (tracking), etc. are monitored. A disturbance beingassociated with when an object is detected.

Thereafter in block 906, one or more proximity gestures are identifiedbased on the characteristics of the disturbances, i.e., the amount oftime the finger hovers, the number of taps, the length of time betweentaps, etc.

Thereafter in block 908, one or more actions are performed based on theidentified proximity gestures.

The proximity gestures may be similar to touch gestures except that notouch occurs. The finger or hand gesture is performed above the surfaceand thus there is no contact. Some examples of touch gestures that maybe applied to proximity gestures can be found in U.S. patent applicationSer. Nos. 10/903,964 and 11/038,590, which are herein incorporated byreference.

FIG. 23 is a block diagram of an exemplary computer system 950, inaccordance with one embodiment of the present invention. The computersystem 950 may correspond to a personal computer, such as a desktop,laptop, tablet or handheld computer. The computer system may alsocorrespond to other types of computing devices such as a cell phones,PDAs, media players, consumer electronic devices, and/or the like.

The exemplary computer system 950 shown in FIG. 23 includes a processor956 configured to execute instructions and to carry out operationsassociated with the computer system 950. For example, using instructionsretrieved for example from memory, the processor 956 may control thereception and manipulation of input and output data between componentsof the computing system 950. The processor 956 can be implemented on asingle-chip, multiple chips or multiple electrical components. Forexample, various architectures can be used for the processor 956,including dedicated or embedded processor, single purpose processor,controller, ASIC, and so forth.

In most cases, the processor 956 together with an operating systemoperates to execute computer code and produce and use data. By way ofexample, the operating system may correspond to Mac OS, OS/2, DOS, Unix,Linux, Palm OS, and the like. The operating system can also be a specialpurpose operating system, such as may be used for limited purposeappliance-type computing devices. The operating system, other computercode and data may reside within a memory block 658 that is operativelycoupled to the processor 656. Memory block 658 generally provides aplace to store computer code and data that are used by the computersystem 950. By way of example, the memory block 658 may includeRead-Only Memory (ROM), Random-Access Memory (RAM), hard disk driveand/or the like. The information could also reside on a removablestorage medium and loaded or installed onto the computer system 650 whenneeded. Removable storage media include, for example, CD-ROM, PC-CARD,memory card, floppy disk, magnetic tape, and a network component.

The computer system 950 also includes a display device 968 that isoperatively coupled to the processor 956. The display device 968 may bea liquid crystal display (LCD) (e.g., active matrix, passive matrix andthe like). Alternatively, the display device 968 may be a monitor suchas a monochrome display, color graphics adapter (CGA) display, enhancedgraphics adapter (EGA) display, variable-graphics-array (VGA) display,super VGA display, cathode ray tube (CRT), and the like. The displaydevice may also correspond to a plasma display or a display implementedwith electronic inks or OLEDs.

The display device 968 is generally configured to display a graphicaluser interface (GUI) that provides an easy to use interface between auser of the computer system and the operating system or applicationrunning thereon. Generally speaking, the GUI represents, programs, filesand operational options with graphical images. The graphical images mayinclude windows, fields, dialog boxes, menus, icons, buttons, cursors,scroll bars, etc. Such images may be arranged in predefined layouts, ormay be created dynamically to serve the specific actions being taken bya user. During operation, the user can select and activate variousgraphical images in order to initiate functions and tasks associatedtherewith. By way of example, a user may select a button that opens,closes, minimizes, or maximizes a window, or an icon that launches aparticular program. The GUI can additionally or alternatively displayinformation, such as non interactive text and graphics, for the user onthe display device 968.

The computer system 950 also includes an input device 970 that isoperatively coupled to the processor 956. The input device 970 isconfigured to transfer data from the outside world into the computersystem 950. The input device 970 may include a touch sensing deviceconfigured to receive input from a user's touch and to send thisinformation to the processor 956. In many cases, the touch-sensingdevice recognizes touches, as well as the position and magnitude oftouches on a touch sensitive surface. The touch sensing means reportsthe touches to the processor 956 and the processor 956 interprets thetouches in accordance with its programming. For example, the processor956 may initiate a task in accordance with a particular touch. Adedicated processor can be used to process touches locally and reducedemand for the main processor of the computer system. The touch sensingdevice may be based on sensing technologies including but not limited tocapacitive sensing, resistive sensing, surface acoustic wave sensing,pressure sensing, optical sensing, and/or the like. Furthermore, thetouch sensing means may be based on single point sensing or multipointsensing. Single point sensing is capable of only distinguishing a singletouch, while multipoint sensing is capable of distinguishing multipletouches that occur at the same time.

In the illustrated embodiment, the input device 970 is a touch screenthat is positioned over or in front of the display 968. The touch screen670 may be integrated with the display device 968 or it may be aseparate component. The touch screen 970 has several advantages overother input technologies such as touchpads, mice, etc. For one, thetouch screen 670 is positioned in front of the display 968 and thereforethe user can manipulate the GUI directly. For example, the user cansimply place their finger over an object to be selected, activated,controlled, etc. In touch pads, there is no one-to-one relationship suchas this. With touchpads, the touchpad is placed away from the displaytypically in a different plane. For example, the display is typicallylocated in a vertical plane and the touchpad is typically located in ahorizontal plane. This makes its use less intuitive, and therefore moredifficult when compared to touch screens.

The touchscreen can be a single point or multipoint touchscreen.Multipoint input devices have advantages over conventional single pointdevices in that they can distinguish more than one object (finger)simultaneously. Single point devices are simply incapable ofdistinguishing multiple objects at the same time. By way of example, amultipoint touch screen, which can be used herein, is shown anddescribed in greater detail in copending and commonly assigned U.S.patent application Ser. No. 10/840,862, which is hereby incorporatedherein by reference.

The computer system 950 also includes a proximity detection system 990that is operatively coupled to the processor 956. The proximitydetection system 990 is configured to detect when a finger (or stylus)is in close proximity to (but not in contact with) some component of thecomputer system including for example housing or I/O devices such as thedisplay and touch screen. The proximity detection system 990 may bewidely varied. For example, it may be based on sensing technologiesincluding capacitive, electric field, inductive, hall effect, reed, eddycurrent, magneto resistive, optical shadow, optical visual light,optical IR, optical color recognition, ultrasonic, acoustic emission,radar, heat, sonar, conductive or resistive and the like. A few of thesetechnologies will now be briefly described.

The computer system 950 also includes capabilities for coupling to oneor more I/O devices 980. By way of example, the I/O devices 980 maycorrespond to keyboards, printers, scanners, cameras, speakers, and/orthe like. The I/O devices 980 may be integrated with the computer system950 or they may be separate components (e.g., peripheral devices). Insome cases, the I/O devices 980 may be connected to the computer system950 through wired connections (e.g., cables/ports). In other cases, theI/O devices 980 may be connected to the computer system 950 throughwireless connections. By way of example, the data link may correspond toPS/2, USB, IR, RF, Bluetooth or the like.

While this invention has been described in terms of several preferredembodiments, there are alterations, permutations, and equivalents, whichfall within the scope of this invention. It should also be noted thatthere are many alternative ways of implementing the methods andapparatuses of the present invention. It is therefore intended that thefollowing appended claims be interpreted as including all suchalterations, permutations, and equivalents as fall within the truespirit and scope of the present invention.

The invention claimed is:
 1. A method comprising: at an electronicdevice with a touch screen: while displaying, on the touch screen, auser interface, detecting, via the touch screen, a hover input thatincludes detecting an object over the touch screen without detecting theobject in contact with the touch screen; and in response to detectingthe hover input: in accordance with a determination that one or morecriteria are met, the one or more criteria including a criterion thatrequires that the hover input be detected for at least a predeterminedtime threshold, displaying, on the touch screen, a control correspondingto the user interface; and in accordance with a determination that theone or more criteria are not met, forgoing displaying the control on thetouch screen.
 2. The method of claim 1, wherein the one or more criteriainclude a criterion that requires that the hover input be detected whilea predefined application is displayed, a criterion that requires thehover input be detected while the predefined application be in aspecified state or mode, a criterion that requires the hover inputinclude a specified number of objects detected over the touch screen, ora criterion that requires the hover input be detected at a specifiedlocation on the touch screen.
 3. The method of claim 1, whereindisplaying the control comprises gradually displaying the control on thetouch screen by increasing a visual property of the control over time.4. The method of claim 3, wherein a rate or amount of graduallydisplaying the control is based on a length of time the hover input isdetected on the touch screen.
 5. The method of claim 1, wherein thecontrol is displayed at a location on the touch screen based on alocation of the hover input detected at the touch screen.
 6. The methodof claim 1, wherein the control includes a first portion and a secondportion, the method further comprising: while displaying the control anddetecting the hover input: in accordance with a determination that thehover input is over the first portion, enlarging the first portion; andin accordance with a determination that the object is over the secondportion, enlarging the second portion.
 7. The method of claim 1, furthercomprising: after detecting the hovering input and while continuing todisplay the control in the user interface on the touch screen,detecting, via the touch screen, the object touching the touch screen;and in response to detecting the object touching the touch screen: inaccordance with a determination that the object touches the touch screenat a respective location at which the control is displayed on the touchscreen, performing an action corresponding to the control; and inaccordance with a determination that the object touches the touch screenat the respective location at which the control is not displayed,forgoing performing the action corresponding to the control.
 8. Themethod of claim 1, further comprising: while displaying the control onthe touch screen, determining that the one or more criteria are nolonger met; and in response to determining that the one or more firstcriteria are no longer met, ceasing displaying the control on the touchscreen.
 9. The method of claim 8, wherein ceasing display of the controlcomprises gradually removing the control from the display.
 10. Themethod of claim 9, wherein determining that the one or more criteria areno longer met includes determining that the hover input is no longerdetected at the touch screen for at least a second predetermined timethreshold.
 11. The method of claim 8, wherein determining that the oneor more criteria are no longer met includes determining that the hoverinput is no longer detected at the touch screen, determining that a thecontrol has been displayed for a second predetermined time threshold, ordetermining that the control has been selected.
 12. The method of claim1, wherein input to the control manipulates content displayed in theuser interface.
 13. The method of claim 1, wherein input to the controlnavigates forward and backward through content displayed in the userinterface.
 14. The method of claim 1, further comprising: afterdetecting the hover input, detecting, via the touch screen, a secondhover input; and in response to detecting the second hover input: inaccordance with a determination that the one or more criteria are metfor the second hover input, displaying, on the touch screen, a secondcontrol, different from the control; and in accordance with adetermination that the one or more criteria are not met for the secondhover input, maintaining display of the control.
 15. The method of claim1, further comprising: while detecting the hover input, detecting, viathe touch screen, a second hover input; and in response to detecting thesecond hover input: in accordance with a determination that the one ormore criteria are met for the second hover input, displaying, on thetouch screen, a second control corresponding to the user interface whilemaintaining display of the control; and in accordance with adetermination that the one or more criteria are not met for the secondhover input, maintaining display of the control without displaying thesecond control.
 16. An electronic device, comprising: a touchscreen oneor more processors; memory; and one or more programs, wherein the one ormore programs are stored in the memory and configured to be executed bythe one or more processors, the one or more programs includinginstructions for: while displaying, on the touch screen, a userinterface, detecting, via the touch screen, a hover input that includesdetecting an object over the touch screen without detecting the objectin contact with the touch screen; and in response to detecting the hoverinput: in accordance with a determination that one or more criteria aremet, the one or more criteria including a criterion that requires thatthe hover input be detected for at least a predetermined time threshold,displaying, on the touch screen, a control corresponding to the userinterface; and in accordance with a determination that the one or morecriteria are not met, forgoing displaying the control on the touchscreen.
 17. The electronic device of claim 16, wherein the controlincludes a first portion and a second portion, the one or more programsfurther including instructions for: while displaying the control anddetecting the hover input: in accordance with a determination that thehover input is over the first portion, enlarging the first portion; andin accordance with a determination that the object is over the secondportion, enlarging the second portion.
 18. The electronic device ofclaim 16, the one or more programs further including instructions for:while displaying the control on the touch screen, determining that theone or more criteria are no longer met; and in response to determiningthat the one or more first criteria are no longer met, ceasingdisplaying the control on the touch screen.
 19. The electronic device ofclaim 16, the one or more programs further including instructions for:after detecting the hover input, detecting, via the touch screen, asecond hover input; and in response to detecting the second hover input:in accordance with a determination that the one or more criteria are metfor the second hover input, displaying, on the touch screen, a secondcontrol, different from the control; and in accordance with adetermination that the one or more criteria are not met for the secondhover input, maintaining display of the control.
 20. The electronicdevice of claim 16, the one or more programs further includinginstructions for: while detecting the hover input, detecting, via thetouch screen, a second hover input; and in response to detecting thesecond hover input: in accordance with a determination that the one ormore criteria are met for the second hover input, displaying, on thetouch screen, a second control corresponding to the user interface whilemaintaining display of the control; and in accordance with adetermination that the one or more criteria are not met for the secondhover input, maintaining display of the control without displaying thesecond control.
 21. A non-transitory computer readable storage mediumstoring one or more programs, the one or more programs comprisinginstructions, which when executed by one or more processors of anelectronic device including a touch screen, cause the electronic deviceto: while displaying, on the touch screen, a user interface, detect, viathe touch screen, a hover input that includes detecting an object overthe touch screen without detecting the object in contact with the touchscreen; and in response to detecting the hover input: in accordance witha determination that one or more criteria are met, the one or morecriteria including a criterion that requires that the hover input bedetected for at least a predetermined time threshold, display, on thetouch screen, a control corresponding to the user interface; and inaccordance with a determination that the one or more criteria are notmet, forgo displaying the control on the touch screen.
 22. Thenon-transitory computer readable storage medium of claim 21, wherein thecontrol includes a first portion and a second portion, the one or moreprograms further comprising instructions which when executed cause theelectronic device to: while displaying the control and detecting thehover input: in accordance with a determination that the hover input isover the first portion, enlarge the first portion; and in accordancewith a determination that the object is over the second portion, enlargethe second portion.
 23. The non-transitory computer readable storagemedium of claim 21, the one or more programs further comprisinginstructions which when executed cause the electronic device to: whiledisplaying the control on the touch screen, determine that the one ormore criteria are no longer met; and in response to determining that theone or more first criteria are no longer met, cease displaying thecontrol on the touch screen.
 24. The non-transitory computer readablestorage medium of claim 21, the one or more programs further comprisinginstructions which when executed cause the electronic device to: afterdetecting the hover input, detect, via the touch screen, a second hoverinput; and in response to detecting the second hover input: inaccordance with a determination that the one or more criteria are metfor the second hover input, display, on the touch screen, a secondcontrol, different from the control; and in accordance with adetermination that the one or more criteria are not met for the secondhover input, maintain display of the control.
 25. The non-transitorycomputer readable storage medium of claim 21, the one or more programsfurther comprising instructions which when executed cause the electronicdevice to: while detecting the hover input, detect, via the touchscreen, a second hover input; and in response to detecting the secondhover input: in accordance with a determination that the one or morecriteria are met for the second hover input, display, on the touchscreen, a second control corresponding to the user interface whilemaintaining display of the control; and in accordance with adetermination that the one or more criteria are not met for the secondhover input, maintain display of the control without displaying thesecond control.
 26. The non-transitory computer readable storage mediumof claim 21, wherein the one or more criteria include a criterion thatrequires that the hover input be detected while a predefined applicationis displayed, a criterion that requires the hover input be detectedwhile the predefined application be in a specified state or mode, acriterion that requires the hover input include a specified number ofobjects detected over the touch screen, or a criterion that requires thehover input be detected at a specified location on the touch screen. 27.The non-transitory computer readable storage medium of claim 21, whereindisplaying the control comprises gradually displaying the control on thetouch screen by increasing a visual property of the control over time.28. The non-transitory computer readable storage medium of claim 27,wherein a rate or amount of gradually displaying the control is based ona length of time the hover input is detected on the touch screen. 29.The non-transitory computer readable storage medium of claim 21, whereinthe control is displayed at a location on the touch screen based on alocation of the hover input detected at the touch screen.
 30. Thenon-transitory computer readable storage medium of claim 21, the one ormore programs further comprising instructions which when executed causethe electronic device to: after detecting the hovering input and whilecontinuing to display the control in the user interface on the touchscreen, detect, via the touch screen, an object touching the touchscreen; and in response to detecting the object touching the touchscreen: in accordance with a determination that the object touches thetouch screen at a respective location at which the control is displayedon the touch screen, perform an action corresponding to the control; andin accordance with a determination that the object touches the touchscreen at the respective location at which the control is not displayed,forgo performing the action corresponding to the control.
 31. Thenon-transitory computer readable storage medium of claim 23, whereinceasing display of the control comprises gradually removing the controlfrom the display.
 32. The non-transitory computer readable storagemedium of claim 31, wherein determining that the one or more criteriaare no longer met includes determining that the hover input is no longerdetected at the touch screen for at least a second predetermined timethreshold.
 33. The non-transitory computer readable storage medium ofclaim 23, wherein determining that the one or more criteria are nolonger met includes determining that the hover input is no longerdetected at the touch screen, determining that a the control has beendisplayed for a second predetermined time threshold, or determining thatthe control has been selected.
 34. The non-transitory computer readablestorage medium of claim 21, wherein input to the control manipulatescontent displayed in the user interface.
 35. The non-transitory computerreadable storage medium of claim 21, wherein input to the controlnavigates forward and backward through content displayed in the userinterface.
 36. The non-transitory computer readable storage medium ofclaim 21, the one or more programs further comprising instructions whichwhen executed cause the electronic device to: after detecting the hoverinput, while displaying, on the touch screen, the user interface,detect, via the touch screen, an input; and in response to detecting theinput: in accordance with a determination that the input is a secondhover input corresponding to the object over the touch screen withoutdetecting the object in contact with the touch screen: in accordancewith a determination that the one or more criteria are met, the one ormore criteria including the criterion that requires that the secondhover input be detected for at least the predetermined time threshold,display, on the touch screen, a second control corresponding to the userinterface; and in accordance with a determination that the one or morecriteria are not met, forgo displaying the second control on the touchscreen; and in accordance with a determination that the input is a touchinput corresponding to the object touching the touch screen, perform anaction corresponding to the touch input, wherein the action is differentthan displaying the second control on the touch screen.
 37. The methodof claim 1, further comprising: after detecting the hover input, whiledisplaying, on the touch screen, the user interface, detecting, via thetouch screen, an input; and in response to detecting the input: inaccordance with a determination that the input is a second hover inputcorresponding to the object over the touch screen without detecting theobject in contact with the touch screen: in accordance with adetermination that the one or more criteria are met, the one or morecriteria including the criterion that requires that the second hoverinput be detected for at least the predetermined time threshold,displaying, on the touch screen, a second control corresponding to theuser interface; and in accordance with a determination that the one ormore criteria are not met, forgoing displaying the second control on thetouch screen; and in accordance with a determination that the input is atouch input corresponding to the object touching the touch screen,performing an action corresponding to the touch input, wherein theaction is different than displaying the second control on the touchscreen.
 38. The electronic device of claim 16, wherein the one or morecriteria include a criterion that requires that the hover input bedetected while a predefined application is displayed, a criterion thatrequires the hover input be detected while the predefined application bein a specified state or mode, a criterion that requires the hover inputinclude a specified number of objects detected over the touch screen, ora criterion that requires the hover input be detected at a specifiedlocation on the touch screen.
 39. The electronic device of claim 16,wherein displaying the control comprises gradually displaying thecontrol on the touch screen by increasing a visual property of thecontrol over time.
 40. The electronic device of claim 39, wherein a rateor amount of gradually displaying the control is based on a length oftime the hover input is detected on the touch screen.
 41. The electronicdevice of claim 16, wherein the control is displayed at a location onthe touch screen based on a location of the hover input detected at thetouch screen.
 42. The electronic device of claim 16, the one or moreprograms further including instructions for: after detecting thehovering input and while continuing to display the control in the userinterface on the touch screen, detecting, via the touch screen, anobject touching the touch screen; and in response to detecting theobject touching the touch screen: in accordance with a determinationthat the object touches the touch screen at a respective location atwhich the control is displayed on the touch screen, performing an actioncorresponding to the control; and in accordance with a determinationthat the object touches the touch screen at the respective location atwhich the control is not displayed, forgoing performing the actioncorresponding to the control.
 43. The electronic device of claim 18,wherein ceasing display of the control comprises gradually removing thecontrol from the display.
 44. The electronic device of claim 43, whereindetermining that the one or more criteria are no longer met includesdetermining that the hover input is no longer detected at the touchscreen for at least a second predetermined time threshold.
 45. Theelectronic device of claim 18, wherein determining that the one or morecriteria are no longer met includes determining that the hover input isno longer detected at the touch screen, determining that a the controlhas been displayed for a second predetermined time threshold, ordetermining that the control has been selected.
 46. The electronicdevice of claim 16, wherein input to the control manipulates contentdisplayed in the user interface.
 47. The electronic device of claim 16,wherein input to the control navigates forward and backward throughcontent displayed in the user interface.
 48. The electronic device ofclaim 16, the one or more programs further including instructions for:after detecting the hover input, while displaying, on the touch screen,the user interface, detecting, via the touch screen, an input; and inresponse to detecting the input: in accordance with a determination thatthe input is a second hover input corresponding to the object over thetouch screen without detecting the object in contact with the touchscreen: in accordance with a determination that the one or more criteriaare met, the one or more criteria including the criterion that requiresthat the second hover input be detected for at least the predeterminedtime threshold, displaying, on the touch screen, a second controlcorresponding to the user interface; and in accordance with adetermination that the one or more criteria are not met, forgoingdisplaying the second control on the touch screen; and in accordancewith a determination that the input is a touch input corresponding tothe object touching the touch screen, performing an action correspondingto the touch input, wherein the action is different than displaying thesecond control on the touch screen.